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BACKGROUND 

frigid of ^ hg> Tnvention 

The present invention relates to image 
stabilization. More particularly, this invention 
pertains to a method for image sequence stabilization 
that includes systematic elimination of blank areas in 
frames in the presence of image location adjustments. 

Videos, especially those taken by hand-held 
means, such as home videos taken by amateur 
videographers, often include "shaky" portions due to 
the presence of, for example, unintentional hand tremor 
or other platform instability. 

Prior solutions to the problem of filtering 
unwanted motion from videos have involved the 
relocation of images, effectively smoothing the overall 
motion. Commonly, a particular, perhaps prominent, 
feature, is selected and its location (horizontal, 
vertical or both) tracked from f rame- to- frame . 
Thereafter, a filtering-based algorithm is applied to 
generate frame-by- frame adjustments of the (horizontal, 
vertical, or both) location of each image that includes 
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such selected feature within or with respect to its 
frame • 

While reducing the effect of such factors as 
hand tremor and smoothing motion to present a more- 
professional appearance, such techniques are subject to 
the creation of often-distracting blank border areas 
that differ from frame-to-frame. This can result in 
annoying "boundary flicker", reflecting the fact that 
the undesired hand tremor has been effectively 
transported from the image feature to the frame's 
borders by the step of motion compensation resulting in 
image relocation within the frame pursuant to the 
selected algorithm. 

SUMMARY OF THE INVENTION 
The preceding and other shortcomings of the 
prior art are addressed by the present invention that 
provides, in a first aspect, a method for removing 
undesired motion from a sequential plurality of related 
images each of which includes a stationary component. 
Each image is fixed within a frame of uniform 
dimension . 

A plurality of sequential inter-frame 
movements of the component is determined over the 
plurality of frames, each inter-frame movement being 
associated with one of the frames. A reference frame 
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is chosen that has an associated inter-frame movement. 
The difference between each of said inter-frame 
movements and the inter-frame movement associated wrth 
the reference frame is measured. 

An image shift parameter, a smoothing filter 
and a test frame are chosen. At least one additional 
frame is chosen adjacent the test frame, the frames 
defining a sample. 

The smoothing filter is applied to the 
difference values of the sample to obtain a filtered 
difference value from which the difference associated 
with the test frame is subtracted to obtain an image 
shift value. 

The image shift value is compared to the 

image shift parameter and the number of frames of the 

sample are adjusted in accordance with such comparison 

oMft value is re-calculated on the basis of 
The image shirt vaiue is 

the adjusted number of frames of the sample. 

The position of the image is shifted with 
respect to the test frame by the image shift value. 

The next frame of the sequence is then 
designated as the test frame. 



in a second aspect, the invention provides a 
method for removing undesired motion from a seguentral 
plurality of related images each of which includes a 
stationary component. Each image is fixed within a 
frame of uniform dimension. 

A plurality of sequential inter-frame 
raovements of the component is determined over the 
plurality of frames, each inter-frame movement being 
associated with one of the frames. 

A reference frame is chosen that has an 
associated inter-frame movement. The difference 
between each of said inter-frame movements and the 
inter-frame movement associated with the reference 
frame is measured. 

An image shift parameter, a smoothing filter 
and a test frame are chosen. At least one additional 
frame is chosen adjacent the test frame, the frames 
defining a sample. 

The smoothing filter is applied to the 
difference value of the sample to obtain a filtered 
difference value from which the difference associated 
with the test frame is subtracted to obtain an image 
shift value. The position of the image is shifted with 
respect to the test frame by the image shift value. 



The next frame of the sequence is then 
designated as the test frame. Test frames are 
continued to be designated from the sequence until each, 
frame of the sequence has been designated and the image 
fixed thereon shifted in accordance with the above- 
described process. All of the frames of the sequence 
are then enlarged by a zoom factor related to the image 
shift parameter. 

The foregoing and other features of this 
invention will become further apparent from the 
detailed description that follows. Such description is 
accompanied by a set of drawing figures. Numerals of 
the written description, corresponding to those of the 
drawing figures, point to the features of the invention 
with like numerals referring to like features 
throughout both the written description and the 
drawings . 

BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 is a flow chart of a method for 
image sequence stabilization in accordance with an 
embodiment of the invention; and 
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Figures 2(a) through 2(c) is a series of 
graphs that illustrate the process of image 
stabilization in accordance with an embodiment of the 
invention . 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
Figure 1 is a flow chart of a method for 
frame-to-frame image motion stabilization in accordance, 
with an embodiment of the invention. The method 
employs an adaptive filtering technique to make 
adjustments to the positions of successive images with 
respect to frames of uniform dimension. Thereafter, a 
preselected enlargement or zoom factor is applied to 
all images, effectively filling the frames with the 
images to eliminate boundary flicker. As will be seen, 
by employing an adaptive filtering technique that is 
integrated with an image shift parameter (related to 
the zoom factor), the method of the invention attains 
the benefits of improved image sequence stability and 
absence of boundary flicker at an acceptable resolution 
level . 

. I 

In discussing the method of the invention 
with reference to the flow chart of Figure 1, it is 
assumed that the effects of undesired camera motion are 
limited to and measurable, by the displacement of a 
stationary component of the image (e.g. a tree in the 
background) only with respect to the ordinate or "y" 
axis. Such an assumption is made for the sole purpose 
of facilitating and clarifying the discussion that 
follows. However, it will be appreciated that the 
discussion may be extended to include and correct for 
undesired image motion with respect to multiple axes 
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(i.e. including "x" axis motion in an x-y reference 
system as well as undesired motions defined with 
respect to axes of other reference systems) . 
The position of the stationary component in the i-th 
frame is designated yi . 

The method of the invention is begun at step 
S-l by computing the sequential inter-frame movements 
Ay± (= Yi ~ Yi-i) of the stationary component of the 
image over the sequence of frames. Such values 
represent the sequential motion of the image sensor 
(e.g. camera) as opposed to actual motion of the 
component and will typically be measured in pixels of 
the digital or digitized images captured. Computation 
of the Ay ± values may be accomplished by means of a 
number of algorithms that are capable of selecting a 
stationary image component and measuring a 
corresponding sequence of motions. See, for example, 
Hapreet S. Sawhney and Serge Ayer, "Compact 
Representations of Videos Through Dominant and Multiple 
Motion Estimation," IEEE Transactions on Pattern 
Analysis and Machine Intelligence, vol. 18, no. 8 
(1996), pp. 814-830 and J. M. Odobez and P. Bouthemy, 
"Robust Multiresolution Estimation of Parametric Motion 
Models," Journal of Visual communication and Image 
Representation, vol. 6, no. 4 (December 1995), pp. 348- 
365. The measurement of Ay ± may be based upon the 
assumption of simple transnational motion or a more 
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complex motion model, such as affine motion that takes 
rotation, scaling and shearing into account. 

At step S-2, a single frame of the_ image 
sequence, with an associated inter-frame movement value 
Ay re fr is selected and is designated, for purposes of 
the discussion that follows, as the "reference frame". 
Such a selection maybe made arbitrarily although the 
first frame of the sequence is generally selected for 
convenience . 

A value of Ayi having been established for 
each frame, including the selected reference frame, at 
step S-l and a reference frame designated at step S-2, 
another sequence of values AYi (= Ay A - Ay ref ) is 
calculated at step S-3. Each value of AY t is associated 
with a frame i, just as a value of Ay t is associated 
with each frame. 

The AYi values provide indications of the 
degree to which the selected stationary component in 
any of the (i - 1) remaining frames is displaced 

relative to an established reference. This, unlike a 

i 

Ayi value, provides a basis | for stabilizing the full 
sequence with respect to a common reference frame. 

An image shift parameter 6 is selected at 
step S-4. While * is theoretically capable of being 



Hg PD 10P205024-1 

assigned any value, in practical application its choice 
is user or application dependent. As will be seen 
below, boundary flicker is removed, after image 
shifting for the purpose of motion stabilization, by 
enlarging the images to fill otherwise-blank border 
portions of associated frames. Such enlargement or 
zooming of an image reduces image resolution. 
Preferably, the value of 6 will dictate * represents 
an amount of image enlargement or zoom factor that 
results in, at most, a modest, and preferably 
imperceptible, reduction in resolution. An acceptable 
reduction of image resolution is a function of the 
video application and will vary in accordance with the 
video quality required, for example, by an audience. 

Once a sequence of values of AYi has been 
derived and an image shift parameter 5 chosen, an 
adaptive filtering process is employed to determine 
adjustments to the locations or positions of the 
individual images (y) of the image sequence with 
respect to their associated frames (i) . The values of 
the adjustments represent the amounts (generally 
measured in pixels) that individual images are to be 
shifted to remove undesired motion from the image 
sequence. 

The adaptive filtering process described 
below is designed to assure that the image shifts are 
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determined on the basis of statistical samples (i.e., 
number of images) of maximum size. In this way, 
maximum sequence motion smoothness (stability) is 
obtained. Each sample comprises the test image plus at 
least one other image that is adjacent the test image. 
The adaptive filtering process described below yields 
sample sizes that are inversely related (although not 
in the strict sense) to the amount of motion (position 
shifting) of the stationary component in the test frame 
relative to that of the stationary component in 
adjacent frames. Thus, sudden or jerky motions are 
compensated in a relatively isolated manner with few 
adjacent frames comprising the filtered sample. In 
this way, the intended dramatic or large shift of the 
image with respect to the test frame is not masked by 
the inclusion of data covering more tranquil segments 
of the image sequence. On the other hand, the frame- 
to-frame smoothness of image shifting is enhanced when 
based upon large sample sizes during more tranquil 
segments of the image sequence. 

The adaptive filtering process involves the 
computation of successive frame-by-frame image 
adjustments or shifts. Logically, although not 
exclusively, such method will proceed sequentially 
through all frames of the image sequence, with the 
first frame chosen as the reference frame. 
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Turning to the adaptive filtering process, a 
representative (i-th) test frame is selected for 
determination of an appropriate image shift at step S- . 
5. At least one additional image of the image sequence 
adjacent the test image is chosen, the test image and 
the additional adjacent image (s) forming a sample, at 
step S-6. The images of the image sequence forming the 
sample may be arranged either symmetrically or 
asymmetrically with respect to the test frame. The 
type of sample chosen will depend upon the type of a 
smoothing filter F() to be applied. Examples of 
smoothing filters include, among others, Gaussian, 
Kalman and box filters. In' contrast, non-smoothing 
filters include, but are not limited to, median and 
edge detector filters. In the case of a symmetric 
filter (e.g. a Gaussian filter), images are paired on 
either side of the test image creating samples 
comprising an odd number of images while appropriate 
samples for application to an asymmetric smoothing 
filter (e.g. a Kalman filter) may comprise either an 
even or odd number of images . 

The selected smoothing filter is applied to 
the AYi values of the images of the selected sample at 
step S-7. The AYi value of' the test image is then 
subtracted from the filtered value of the sample, 
EF(AYi). The resultant difference comprises an image 
shift value that is then compared to the image shift 
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parameter 5. In the event such difference is 
determined to be less than 5 at step S-8, the size of 
the sample is increased at step S-9 in either a 
symmetrical or asymmetrical manner in accordance with 
the type of filter F() employed and the new, larger 
sample, examined by again cycling through steps S-7 and 
S-8 . 

Once it is determined that the sample is of 
sufficient size or width to yield an image shift value 
(EF(AYi) - Ayi) that is not less than the image shift 
parameter 5, such difference is examined at step S-10 
to determine whether, in perhaps widening the sample 
size, the difference now exceeds 5. Obviously, only for 
the rare case of a sample size that yields an image 
shift value that is exactly equal to the image shift 
parameter 5 will the response to the inquiry at step S- 
10 (which follows the inquiry at step S-8) be negative. 
Rather, in most cases, the size of the sample will be 
decreased in either a symmetrical or asymmetrical 
manner at step S-ll. Thus, although a sample of .maximum 
size is desired (steps S-8, S-9) to obtain maximum 
smoothness, this is tempered (steps S-10, S-ll) by the 
necessity of assuring that ,the image shift parameter 5 
is not exceeded which, as will be seen below, will 
assure that border flicker is removed upon application 
of a zoom factor to all frames of the image sequence 
that is related to the value of the parameter 5 

in 
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(relationship discussed below) . The new sample is 
applied to the filter at step S-12 and again examined 
at step S-10. 

Once the sample size has been determined not 
to be excessive by means of the adaptive filtering 
process described above, the image is shifted by the 
resultant image shift value (SF(AYi) - Y t ) at step S-13. 
This shift represents a correction of the position of 
the image of the test frame for the amount (number of 
pixels) that the stationary component in the test frame 
is out of alignment with the filtered position of such 
component based upon the maximum number of surrounding 
(adjacent) images. 

Once an indication is obtained at step S-14 
that all images of the sequence have been shifted by 
amounts determined as above, all images are enlarged by 
a zoom factor related to the image shift parameter 5, 
increasing their dimensions to exceed those of their 
frames at step S-15. The enlargement of an imag.e 
offsets the blank space created adjacent a boundary 
between an edge of the image and that of its frame by 
shifting of the image with respect to the frame. The 
amount of enlargement, commonly designated as zoom 
factor Z, is related to the image shift parameter 6 for 
a frame of dimension f (in the example employed to 
illustrate the method of the invention, the dimension f 
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is measured along the y axis) by the following 
relationship : 



Z = (1 + 25/f) (Equation 1) 

As each frame is of fixed, uniform size, this 
effectively "fills" each of the frames to eliminate 
boundary flicker from the image sequence. 

Figures 2(a) through 2(c) are a series of 
graphs that illustrate the above-described process of 
image sequence stabilization in accordance with an 
embodiment of the invention. In Figure 2(a) the path 
of a stationary component in a sequence of images is 
plotted along a path 10. Figure 2(b) illustrates the 
bounds, indicated by the lines 12 and 12', 
corresponding to the image shift parameter 5 that 
defines the amount of permissible shifting of 
individual images of the image sequence. 

Figure 2(c) illustrates the general 
relationship between the degree of image sequence 
stability and the size or width of the image samples 14 
through 24 determined in accordance with the adaptive 
filtering process employed 'in accordance with an 
embodiment of the invention. As can be seen, the size 
(number of adjacent images) of a sample is inversely 
related to the localized degree of frame-to-frame 
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motion. Thus, the samples 16 and 18 are relatively 
small or narrow while encompassing regions of 
relatively large frame-to-frame motion along the path 
10 while the sample 20 is relatively wide, covering a 
relatively steady region of the path 10. 

Thus it is seen that the present invention 
provides a method for removing undesired motion from a 
sequential plurality of related images. By employing 
the teachings of the invention, one may substantially 
eliminate the type of undesired motion that is commonly 
introduced into videos taken by means of hand-held 
cameras and other unstable platforms. 

The invention offers a highly reliable means 
of image sequence stabilization based upon an adaptive 
filtering technique in which sample size is varied in 
accordance with the degree of localized frame-to-frame 
image motion. By employing a predetermined factor in 
conjunction with the adaptive filtering process that 
may later be employed to enlarge images shifted in 
accordance with the adaptive filtering process, one may 
eliminate the problem of boundary flicker without undue 

loss of resolution. j 

! 

While the invention has been disclosed with 
reference to its presently-preferred embodiment, it is 
not limited thereto. Rather, the invention is limited 

16 



HP PD 100205024-1 

only insofar as it is defined by the following set of 
patent claims and includes within its scope all 
equivalents thereof. 
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